Sterescopic (or stereo) images, as is known in the art, are images formed by combining (nonstereoscopic) views of the same scene or object from two or more vantage points or angles. The combination of a plurality of different-vantage views of the same scene or object creates a composite image of that scene or object with the appearance of enhanced depth and dimensionality—i.e., more visual information and perspective regarding the scene or object may be provided by a stereo image combining the visual information provided by a plurality of non-stereo views (which may also be regarded as a visual pair) of the object or scene from various vantage points. In one convenient arrangement for providing stereoscopic images, a stereo image may be formed by simultaneously providing two or more views of the object simultaneously to a common imaging point, which in most cases will be a planar imaging surface. To take a simple example, a stereo image of a person's face might be formed by causing two (non-stereo, or direct) images, one from either side of the person's face, to converge simultaneously at the planar surface of a piece of photographic film (or a video camera lens). Accordingly, a composite stereo image, including data from either side of the face, is formed upon the image point or plane and recorded.
Following recording of this stereo image, it may be processed and stored using a variety of techniques. Of particular interest are computerized techniques for image storage and manipulation. For instance, a stereographic photograph print formed by film may be digitally stored by optically scanning the print (for instance, in line-by-line fashion along a defined horizontal scan axis) into a computer memory. Subsequently, the digital stereoscopic image may be altered, printed out, transmitted, or otherwise manipulated by the computer system.
There are a variety of methods and systems for providing the multiple vantage views or non-stereo sub images that are caused to converge at the image point to form a composite stereo image. Catadioptric systems are optical systems that consist of a combination of mirrors and lenses. Catadioptrics can be used to design stereo sensors that use only a single camera. Although such systems use a variety of different mirror shapes and configurations, the underlying motivation is the same. By using multiple mirrors, scene points can be imaged from two or more viewpoints while using only a single camera. The constituent views being supplied by the two or more viewpoints may be conceptualized as being supplied by corresponding “virtual cameras” at the respective viewpoint vantages.
Single camera stereo has several advantages over traditional two-camera stereo. Because only a single camera and digitizer are used, system parameters such as spectral response, gain, and offset are identical for the stereo pair. In addition, only a single set of internal calibration parameters needs to be determined. Perhaps most important is that single camera stereo simplifies data acquisition by only requiring a single camera and digitizer and no hardware or software for synchronization.
Real-time stereo systems, whether catadioptric or two-camera, require images to be rectified prior to stereo matching. A pair of stereo images is rectified if the epipolar lines are aligned with the scan-lines of the images. When properly aligned, the search for correspondence is simplified and thus real-time performance can be obtained. Once the epipolar geometry of a stereo system is determined, rectifying transformations can be applied to the images When properly aligned, the search for correspondence is simplified and thus real-time performance can be obtained. Once the epipolar geometry of a stereo system is determined, rectifying transformations (provided typically by computer software routines) can be applied to the images. However, rectifying in this manner has two disadvantages for real-time stereo. Applying transformations to the images at run-time is both computationally costly and degrades the stereo data due to the resampling of the images.
An alternative to rectifying the images at run-time is to ensure that the geometry of the stereo image-forming system produces images that are rectified from the outset, and hence to not require any computer-provided software rectification. With two-camera stereo this goal is accomplished by removing any rotation between the two-cameras, aligning the direction of translation with the scan lines of the cameras and using identical internal parameters for the two cameras (a difficult task). The geometric requirements of a rectified catadioptric stereo system are not trivial and are not believed to have been studied to date.
Several researchers have demonstrated the use of both curved and planar mirrors to acquire stereo data with a single camera. Curved mirrors have been primarily used to capture a wide field of view. One of the first uses of curved mirrors suggested a wide field of view stereo system consisting of a conventional camera pointed at two specular spheres (see FIG. 1(a)). A similar system using two convex mirrors, one placed on top of the other, has also been proposed (see FIG. 1(b)). Finally, there have been presented several different catadioptric stereo configuration using a single camera with parabolic, elliptic and hyperbolic mirrors.
Several others have also investigated the use of planar mirrors to design single camera stereo sensors. One proposed sensor uses two planar mirrors connected by a hinge centered in the field of view of the camera. It has also been demonstrated how two mirrors in an arbitrary configuration can be self-calibrated and used for single camera stereo (see FIG. 1(c)). Stereo systems using four planar mirrors have, further, been proposed (see FIG. 1(d). By imaging an object and its mirror reflection, it is known that a stereo image can also be obtained using only a single mirror.
In all of these prior art systems, the stereo images are not rectified, therefore the images must be transformed at run-time prior to stereo matching. One exception is a proposed system described that uses prisms rather than mirrors to acquire rectified stereo images from a single camera. Although prisms are an interesting alternative to mirrors, it is not clear that compact sensors with sufficient baseline can be designed.
Real-time stereo systems, whether catadioptric or two-camera, require images to be rectified prior to stereo matching. A pair of stereo images is rectified if the epipolar lines are aligned with the scan-lines of the images. When properly aligned, the search for correspondence is simplified and thus real-time performance can be obtained. Once the epipolar geometry of a stereo system is determined, rectifying transformations can be applied to the images. However, rectifying in this manner has two disadvantages for real-time stereo. Applying transformations to the images at run-time is both computationally costly and degrades the stereo data due to the resampling of the images.
Thus, it would be desirable to provide a system for single-camera forming of stereoscopic images wherein the constituent non-stereo image views did not require any computer-provided rectifying processing. It would further be desirable to provide a system for real-time stereo imaging that lowers computational demands on computer systems employed in processing, storing, manipulating, and transmitting the stereo images formed thereby. It would be desirable, too, to have sensor geometry allowing for minimization of sensor size. It would likewise be desirable to avoid degradation of stereo image quality by reducing or eliminating the need for resampling of images and consequent data degradation. It would also be desirable to provide a system and method having geometric constraints for the locating of the image point or plane vis à vis the imaged object and one or more mirrors employed in supplying the constituent sub-image views to the image point, such that under those geometric constraints, the stereo image delivered to the image point was inherently rectified by virtue of the system geometry, and did not require additional rectification. It would still further be desirable to provide a stereo imaging system meeting these objectives and having a reasonable degree of error tolerance such that minor imperfections in constructing and applying the stereo sensing system would not defeat the object of obtaining useful stereo images. The prior art does not meet these needs.